{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4cRE8IbIrIV"
      },
      "source": [
        "本文涉及的jupter notebook在[篇章4代码库中](https://github.com/datawhalechina/learn-nlp-with-transformers/tree/main/docs/%E7%AF%87%E7%AB%A04-%E4%BD%BF%E7%94%A8Transformers%E8%A7%A3%E5%86%B3NLP%E4%BB%BB%E5%8A%A1)。\n",
        "\n",
        "如果您正在google的colab中打开这个notebook，您可能需要安装Transformers和🤗Datasets库。将以下命令取消注释即可安装。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MOsHUjgdIrIW"
      },
      "outputs": [],
      "source": [
        "!pip install datasets transformers seqeval"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFASsisvIrIb"
      },
      "source": [
        "如果您正在本地打开这个notebook，请确保您已经进行上述依赖包的安装。您也可以在[这里](https://github.com/huggingface/transformers/tree/master/examples/token-classification)找到本notebook的多GPU分布式训练版本。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NAJMZVtIlY1j"
      },
      "source": [
        "本小节所涉及的模型结构与上一篇章中的BERT基本一致，额外需要学习的是特定任务的数据处理方法和模型训练方法。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEJBSTyZIrIb"
      },
      "source": [
        "# *序列标注（token级的分类问题）*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNe9KPPEIBuy"
      },
      "source": [
        "序列标注，通常也可以看作是token级别的分类问题：对每一个token进行分类。在这个notebook中，我们将展示如何使用[🤗 Transformers](https://github.com/huggingface/transformers)中的transformer模型去做token级别的分类问题。token级别的分类任务通常指的是为为文本中的每一个token预测一个标签结果。下图展示的是一个NER实体名词识别任务。\n",
        "\n",
        "![Widget inference representing the NER task](https://github.com/huggingface/notebooks/blob/master/examples/images/token_classification.png?raw=1)\n",
        "\n",
        "最常见的token级别分类任务:\n",
        "\n",
        "- NER (Named-entity recognition 名词-实体识别) 分辨出文本中的名词和实体 (person人名, organization组织机构名, location地点名...).\n",
        "- POS (Part-of-speech tagging词性标注) 根据语法对token进行词性标注 (noun名词, verb动词, adjective形容词...)\n",
        "- Chunk (Chunking短语组块) 将同一个短语的tokens组块放在一起。\n",
        "\n",
        "对于以上任务，我们将展示如何使用简单的Dataset库加载数据集，同时使用transformer中的`Trainer`接口对预训练模型进行微调。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4RRkXuteIrIh"
      },
      "source": [
        "只要预训练的transformer模型最顶层有一个token分类的神经网络层（比如上一篇章提到的`BertForTokenClassification`）（另外，由于transformer库的tokenizer新特性，可能还需要对应的预训练模型有fast tokenizer这个功能，参考[这个表](https://huggingface.co/transformers/index.html#bigtable)），那么本notebook理论上可以使用各种各样的transformer模型（[模型面板](https://huggingface.co/models)），解决任何token级别的分类任务。\n",
        "\n",
        "如果您所处理的任务有所不同，大概率只需要很小的改动便可以使用本notebook进行处理。同时，您应该根据您的GPU显存来调整微调训练所需要的btach size大小，避免显存溢出。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zVvslsfMIrIh"
      },
      "outputs": [],
      "source": [
        "task = \"ner\" #需要是\"ner\", \"pos\" 或者 \"chunk\"\n",
        "model_checkpoint = \"distilbert-base-uncased\"\n",
        "batch_size = 16"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whPRbBNbIrIl"
      },
      "source": [
        "## 加载数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W7QYTpxXIrIl"
      },
      "source": [
        "\n",
        "我们将会使用[🤗 Datasets](https://github.com/huggingface/datasets)库来加载数据和对应的评测方式。数据加载和评测方式加载只需要简单使用`load_dataset`和`load_metric`即可。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IreSlFmlIrIm"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset, load_metric"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CKx2zKs5IrIq"
      },
      "source": [
        "本notebook中的例子使用的是[CONLL 2003 dataset](https://www.aclweb.org/anthology/W03-0419.pdf)数据集。这个notebook应该可以处理🤗 Datasets库中的任何token分类任务。如果您使用的是您自定义的json/csv文件数据集，您需要查看[数据集文档](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files)来学习如何加载。自定义数据集可能需要在加载属性名字上做一些调整。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s_AY1ATSIrIq"
      },
      "outputs": [],
      "source": [
        "datasets = load_dataset(\"conll2003\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RzfPtOMoIrIu"
      },
      "source": [
        "这个`datasets`对象本身是一种[`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict)数据结构. 对于训练集、验证集和测试集，只需要使用对应的key（train，validation，test）即可得到相应的数据。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GWiVUF0jIrIv",
        "outputId": "1d0b055f-6574-4619-ae4e-1ecbc27d10b9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],\n",
              "        num_rows: 14041\n",
              "    })\n",
              "    validation: Dataset({\n",
              "        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],\n",
              "        num_rows: 3250\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],\n",
              "        num_rows: 3453\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": 6,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y1g-ZyrqIBu2"
      },
      "source": [
        "无论是在训练集、验证机还是测试集中，datasets都包含了一个名为tokens的列（一般来说是将文本切分成了很多词），还包含一个名为label的列，这一列对应这tokens的标注。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3EtYfeHIrIz"
      },
      "source": [
        "给定一个数据切分的key（train、validation或者test）和下标即可查看数据。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X6HrpprwIrIz",
        "outputId": "bdc9d209-7f7d-458c-90c0-af46cb634aa2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'chunk_tags': [11, 21, 11, 12, 21, 22, 11, 12, 0],\n",
              " 'id': '0',\n",
              " 'ner_tags': [3, 0, 7, 0, 0, 0, 7, 0, 0],\n",
              " 'pos_tags': [22, 42, 16, 21, 35, 37, 16, 21, 7],\n",
              " 'tokens': ['EU',\n",
              "  'rejects',\n",
              "  'German',\n",
              "  'call',\n",
              "  'to',\n",
              "  'boycott',\n",
              "  'British',\n",
              "  'lamb',\n",
              "  '.']}"
            ]
          },
          "execution_count": 7,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "datasets[\"train\"][0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mz5rVrvdIBu2"
      },
      "source": [
        "所有的数据标签labels都已经被编码成了整数，可以直接被预训练transformer模型使用。这些整数的编码所对应的实际类别储存在`features`中。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EBircQVEIBu2",
        "outputId": "39246819-5be9-4f12-a820-b4ed2aa95452"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Sequence(feature=ClassLabel(num_classes=9, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'], names_file=None, id=None), length=-1, id=None)"
            ]
          },
          "execution_count": 8,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "datasets[\"train\"].features[f\"ner_tags\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9PPhKEmoIBu3"
      },
      "source": [
        "所以以NER为例，0对应的标签类别是”O“， 1对应的是”B-PER“等等。”O“的意思是没有特别实体（no special entity）。本例包含4种实体类别分别是（PER、ORG、LOC，MISC），每一种实体类别又分别有B-（实体开始的token）前缀和I-（实体中间的token）前缀。\n",
        "\n",
        "- 'PER' for person\n",
        "- 'ORG' for organization\n",
        "- 'LOC' for location\n",
        "- 'MISC' for miscellaneous"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tMugLETLIBu3",
        "outputId": "929219d3-0d22-4100-ed7c-348f474d1b21"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']"
            ]
          },
          "execution_count": 9,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "label_list = datasets[\"train\"].features[f\"{task}_tags\"].feature.names\n",
        "label_list"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WHUmphG3IrI3"
      },
      "source": [
        "为了能够进一步理解数据长什么样子，下面的函数将从数据集里随机选择几个例子进行展示。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i3j8APAoIrI3"
      },
      "outputs": [],
      "source": [
        "from datasets import ClassLabel, Sequence\n",
        "import random\n",
        "import pandas as pd\n",
        "from IPython.display import display, HTML\n",
        "\n",
        "def show_random_elements(dataset, num_examples=10):\n",
        "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
        "    picks = []\n",
        "    for _ in range(num_examples):\n",
        "        pick = random.randint(0, len(dataset)-1)\n",
        "        while pick in picks:\n",
        "            pick = random.randint(0, len(dataset)-1)\n",
        "        picks.append(pick)\n",
        "    \n",
        "    df = pd.DataFrame(dataset[picks])\n",
        "    for column, typ in dataset.features.items():\n",
        "        if isinstance(typ, ClassLabel):\n",
        "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
        "        elif isinstance(typ, Sequence) and isinstance(typ.feature, ClassLabel):\n",
        "            df[column] = df[column].transform(lambda x: [typ.feature.names[i] for i in x])\n",
        "    display(HTML(df.to_html()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 546
        },
        "id": "SZy5tRB_IrI7",
        "outputId": "e89108a9-299a-4eb5-c1e7-5aa411b54c60",
        "scrolled": true
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>tokens</th>\n",
              "      <th>pos_tags</th>\n",
              "      <th>chunk_tags</th>\n",
              "      <th>ner_tags</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2227</td>\n",
              "      <td>[Result, of, a, French, first, division, match, on, Friday, .]</td>\n",
              "      <td>[NN, IN, DT, JJ, JJ, NN, NN, IN, NNP, .]</td>\n",
              "      <td>[B-NP, B-PP, B-NP, I-NP, I-NP, I-NP, I-NP, B-PP, B-NP, O]</td>\n",
              "      <td>[O, O, O, B-MISC, O, O, O, O, O, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2615</td>\n",
              "      <td>[Mid-tier, golds, up, in, heavy, trading, .]</td>\n",
              "      <td>[NN, NNS, IN, IN, JJ, NN, .]</td>\n",
              "      <td>[B-NP, I-NP, B-PP, B-PP, B-NP, I-NP, O]</td>\n",
              "      <td>[O, O, O, O, O, O, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>10256</td>\n",
              "      <td>[Neagle, (, 14-6, ), beat, the, Braves, for, the, third, time, this, season, ,, allowing, two, runs, and, six, hits, in, eight, innings, .]</td>\n",
              "      <td>[NNP, (, CD, ), VB, DT, NNPS, IN, DT, JJ, NN, DT, NN, ,, VBG, CD, NNS, CC, CD, NNS, IN, CD, NN, .]</td>\n",
              "      <td>[B-NP, O, B-NP, O, B-VP, B-NP, I-NP, B-PP, B-NP, I-NP, I-NP, B-NP, I-NP, O, B-VP, B-NP, I-NP, O, B-NP, I-NP, B-PP, B-NP, I-NP, O]</td>\n",
              "      <td>[B-PER, O, O, O, O, O, B-ORG, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>10720</td>\n",
              "      <td>[Hansa, Rostock, 4, 1, 2, 1, 5, 4, 5]</td>\n",
              "      <td>[NNP, NNP, CD, CD, CD, CD, CD, CD, CD]</td>\n",
              "      <td>[B-NP, I-NP, I-NP, I-NP, I-NP, I-NP, I-NP, I-NP, I-NP]</td>\n",
              "      <td>[B-ORG, I-ORG, O, O, O, O, O, O, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7125</td>\n",
              "      <td>[MONTREAL, 70, 59, .543, 11]</td>\n",
              "      <td>[NNP, CD, CD, CD, CD]</td>\n",
              "      <td>[B-NP, I-NP, I-NP, I-NP, I-NP]</td>\n",
              "      <td>[B-ORG, O, O, O, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>3316</td>\n",
              "      <td>[Softbank, Corp, said, on, Friday, that, it, would, procure, $, 900, million, through, the, foreign, exchange, market, by, September, 5, as, part, of, its, acquisition, of, U.S., firm, ,, Kingston, Technology, Co, .]</td>\n",
              "      <td>[NNP, NNP, VBD, IN, NNP, IN, PRP, MD, NN, $, CD, CD, IN, DT, JJ, NN, NN, IN, NNP, CD, IN, NN, IN, PRP$, NN, IN, NNP, NN, ,, NNP, NNP, NNP, .]</td>\n",
              "      <td>[B-NP, I-NP, B-VP, B-PP, B-NP, B-SBAR, B-NP, B-VP, B-NP, I-NP, I-NP, I-NP, B-PP, B-NP, I-NP, I-NP, I-NP, B-PP, B-NP, I-NP, B-PP, B-NP, B-PP, B-NP, I-NP, B-PP, B-NP, I-NP, O, B-NP, I-NP, I-NP, O]</td>\n",
              "      <td>[B-ORG, I-ORG, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-LOC, O, O, B-ORG, I-ORG, I-ORG, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>3923</td>\n",
              "      <td>[Ghent, 3, Aalst, 2]</td>\n",
              "      <td>[NN, CD, NNP, CD]</td>\n",
              "      <td>[B-NP, I-NP, I-NP, I-NP]</td>\n",
              "      <td>[B-ORG, O, B-ORG, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>2776</td>\n",
              "      <td>[The, separatists, ,, who, swept, into, Grozny, on, August, 6, ,, still, control, large, areas, of, the, centre, of, town, ,, and, Russian, soldiers, are, based, at, checkpoints, on, the, approach, roads, .]</td>\n",
              "      <td>[DT, NNS, ,, WP, VBD, IN, NNP, IN, NNP, CD, ,, RB, VBP, JJ, NNS, IN, DT, NN, IN, NN, ,, CC, JJ, NNS, VBP, VBN, IN, NNS, IN, DT, NN, NNS, .]</td>\n",
              "      <td>[B-NP, I-NP, O, B-NP, B-VP, B-PP, B-NP, B-PP, B-NP, I-NP, O, B-ADVP, B-VP, B-NP, I-NP, B-PP, B-NP, I-NP, B-PP, B-NP, O, O, B-NP, I-NP, B-VP, I-VP, B-PP, B-NP, B-PP, B-NP, I-NP, I-NP, O]</td>\n",
              "      <td>[O, O, O, O, O, O, B-LOC, O, O, O, O, O, O, O, O, O, O, O, O, O, O, O, B-MISC, O, O, O, O, O, O, O, O, O, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>1178</td>\n",
              "      <td>[Doctor, Masserigne, Ndiaye, said, medical, staff, were, overwhelmed, with, work, ., \"]</td>\n",
              "      <td>[NNP, NNP, NNP, VBD, JJ, NN, VBD, VBN, IN, NN, ., \"]</td>\n",
              "      <td>[B-NP, I-NP, I-NP, B-VP, B-NP, I-NP, B-VP, I-VP, B-PP, B-NP, O, O]</td>\n",
              "      <td>[O, B-PER, I-PER, O, O, O, O, O, O, O, O, O]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10988</td>\n",
              "      <td>[Reuters, historical, calendar, -, September, 4, .]</td>\n",
              "      <td>[NNP, JJ, NN, :, NNP, CD, .]</td>\n",
              "      <td>[B-NP, I-NP, I-NP, O, B-NP, I-NP, O]</td>\n",
              "      <td>[B-ORG, O, O, O, O, O, O]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "show_random_elements(datasets[\"train\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n9qywopnIrJH"
      },
      "source": [
        "## 预处理数据"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YVx71GdAIrJH"
      },
      "source": [
        "在将数据喂入模型之前，我们需要对数据进行预处理。预处理的工具叫`Tokenizer`。`Tokenizer`首先对输入进行tokenize，然后将tokens转化为预模型中需要对应的token ID，再转化为模型需要的输入格式。\n",
        "\n",
        "为了达到数据预处理的目的，我们使用`AutoTokenizer.from_pretrained`方法实例化我们的tokenizer，这样可以确保：\n",
        "\n",
        "- 我们得到一个与预训练模型一一对应的tokenizer。\n",
        "- 使用指定的模型checkpoint对应的tokenizer的时候，我们也下载了模型需要的词表库vocabulary，准确来说是tokens vocabulary。\n",
        "\n",
        "这个被下载的tokens vocabulary会被缓存起来，从而再次使用的时候不会重新下载。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eXNLu_-nIrJI"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer\n",
        "    \n",
        "tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vl6IidfdIrJK"
      },
      "source": [
        "注意：以下代码要求tokenizer必须是transformers.PreTrainedTokenizerFast类型，因为我们在预处理的时候需要用到fast tokenizer的一些特殊特性（比如多线程快速tokenizer）。\n",
        "\n",
        "几乎所有模型对应的tokenizer都有对应的fast tokenizer。我们可以在[模型tokenizer对应表](https://huggingface.co/transformers/index.html#bigtable)里查看所有预训练模型对应的tokenizer所拥有的特点。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i6oeTW6JIBu5"
      },
      "outputs": [],
      "source": [
        "import transformers\n",
        "assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xg2pzjBjIBu5"
      },
      "source": [
        "在[这里big table of models](https://huggingface.co/transformers/index.html#bigtable)查看模型是否有fast tokenizer。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rowT4iCLIrJK"
      },
      "source": [
        "tokenizer既可以对单个文本进行预处理，也可以对一对文本进行预处理，tokenizer预处理后得到的数据满足预训练模型输入格式"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a5hBlsrHIrJL",
        "outputId": "7f9246d2-9c42-4d3f-d66d-a8d2062c6746"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': [101, 7592, 1010, 2023, 2003, 2028, 6251, 999, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}"
            ]
          },
          "execution_count": 14,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer(\"Hello, this is one sentence!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QGaQgVHlIBu6"
      },
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "poPAMZeZIBu6",
        "outputId": "34def228-2d18-4d5b-b19e-4e3b5ba3bd68"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': [101, 7592, 1010, 2023, 2003, 2028, 6251, 3975, 2046, 2616, 1012, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
            ]
          },
          "execution_count": 15,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenizer([\"Hello\", \",\", \"this\", \"is\", \"one\", \"sentence\", \"split\", \"into\", \"words\", \".\"], is_split_into_words=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lhy-636mIBu6"
      },
      "source": [
        "注意transformer预训练模型在预训练的时候通常使用的是subword，如果我们的文本输入已经被切分成了word，那么这些word还会被我们的tokenizer继续切分。举个例子：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a7ROHe3iIBu7",
        "outputId": "f7649bb4-3102-4106-9078-ff3644422884"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['Germany', \"'s\", 'representative', 'to', 'the', 'European', 'Union', \"'s\", 'veterinary', 'committee', 'Werner', 'Zwingmann', 'said', 'on', 'Wednesday', 'consumers', 'should', 'buy', 'sheepmeat', 'from', 'countries', 'other', 'than', 'Britain', 'until', 'the', 'scientific', 'advice', 'was', 'clearer', '.']\n"
          ]
        }
      ],
      "source": [
        "example = datasets[\"train\"][4]\n",
        "print(example[\"tokens\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pi2g-WxrIBu7",
        "outputId": "55b6ad7f-88c5-4f5e-8da4-04e09f00ca58"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['[CLS]', 'germany', \"'\", 's', 'representative', 'to', 'the', 'european', 'union', \"'\", 's', 'veterinary', 'committee', 'werner', 'z', '##wing', '##mann', 'said', 'on', 'wednesday', 'consumers', 'should', 'buy', 'sheep', '##me', '##at', 'from', 'countries', 'other', 'than', 'britain', 'until', 'the', 'scientific', 'advice', 'was', 'clearer', '.', '[SEP]']\n"
          ]
        }
      ],
      "source": [
        "tokenized_input = tokenizer(example[\"tokens\"], is_split_into_words=True)\n",
        "tokens = tokenizer.convert_ids_to_tokens(tokenized_input[\"input_ids\"])\n",
        "print(tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X7QpHu9TIBu7"
      },
      "source": [
        "单词\"Zwingmann\" 和 \"sheepmeat\"继续被切分成了3个subtokens。\n",
        "\n",
        "由于标注数据通常是在word级别进行标注的，既然word还会被切分成subtokens，那么意味着我们还需要对标注数据进行subtokens的对齐。同时，由于预训练模型输入格式的要求，往往还需要加上一些特殊符号比如： `[CLS]` 和 a `[SEP]`。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PSwyBME-IBu7",
        "outputId": "88622a0d-b301-44a4-caea-735cba0bd7b2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "(31, 39)"
            ]
          },
          "execution_count": 18,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(example[f\"{task}_tags\"]), len(tokenized_input[\"input_ids\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J1SuZ8B1IBu8"
      },
      "source": [
        "tokenizer有一个` `word_ids`方法可以帮助我们解决这个问题。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vW5iV3XpIBu8",
        "outputId": "7aa39188-3d76-4c93-a2ec-d839e8d94b88"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[None, 0, 1, 1, 2, 3, 4, 5, 6, 7, 7, 8, 9, 10, 11, 11, 11, 12, 13, 14, 15, 16, 17, 18, 18, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, None]\n"
          ]
        }
      ],
      "source": [
        "print(tokenized_input.word_ids())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bjftJRgyIBu8"
      },
      "source": [
        "\n",
        "我们可以看到，word_ids将每一个subtokens位置都对应了一个word的下标。比如第1个位置对应第0个word，然后第2、3个位置对应第1个word。特殊字符对应了NOne。有了这个list，我们就能将subtokens和words还有标注的labels对齐啦。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ahv45NF_IBu8",
        "outputId": "71417db7-53dd-4169-cfc2-0aa92bc2910b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "39 39\n"
          ]
        }
      ],
      "source": [
        "word_ids = tokenized_input.word_ids()\n",
        "aligned_labels = [-100 if i is None else example[f\"{task}_tags\"][i] for i in word_ids]\n",
        "print(len(aligned_labels), len(tokenized_input[\"input_ids\"]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wh_PU2VDIBu8"
      },
      "source": [
        "我们通常将特殊字符的label设置为-100，在模型中-100通常会被忽略掉不计算loss。\n",
        "\n",
        "我们有两种对齐label的方式：\n",
        "- 多个subtokens对齐一个word，对齐一个label\n",
        "- 多个subtokens的第一个subtoken对齐word，对齐一个label，其他subtokens直接赋予-100.\n",
        "\n",
        "我们提供这两种方式，通过`label_all_tokens = True`切换。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3-y_VJEIIBu8"
      },
      "outputs": [],
      "source": [
        "label_all_tokens = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2C0hcmp9IrJQ"
      },
      "source": [
        "\n",
        "最后我们将所有内容合起来变成我们的预处理函数。`is_split_into_words=True`在上面已经结束啦。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vc0BSBLIIrJQ"
      },
      "outputs": [],
      "source": [
        "def tokenize_and_align_labels(examples):\n",
        "    tokenized_inputs = tokenizer(examples[\"tokens\"], truncation=True, is_split_into_words=True)\n",
        "\n",
        "    labels = []\n",
        "    for i, label in enumerate(examples[f\"{task}_tags\"]):\n",
        "        word_ids = tokenized_inputs.word_ids(batch_index=i)\n",
        "        previous_word_idx = None\n",
        "        label_ids = []\n",
        "        for word_idx in word_ids:\n",
        "            # Special tokens have a word id that is None. We set the label to -100 so they are automatically\n",
        "            # ignored in the loss function.\n",
        "            if word_idx is None:\n",
        "                label_ids.append(-100)\n",
        "            # We set the label for the first token of each word.\n",
        "            elif word_idx != previous_word_idx:\n",
        "                label_ids.append(label[word_idx])\n",
        "            # For the other tokens in a word, we set the label to either the current label or -100, depending on\n",
        "            # the label_all_tokens flag.\n",
        "            else:\n",
        "                label_ids.append(label[word_idx] if label_all_tokens else -100)\n",
        "            previous_word_idx = word_idx\n",
        "\n",
        "        labels.append(label_ids)\n",
        "\n",
        "    tokenized_inputs[\"labels\"] = labels\n",
        "    return tokenized_inputs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0lm8ozrJIrJR"
      },
      "source": [
        "以上的预处理函数可以处理一个样本，也可以处理多个样本exapmles。如果是处理多个样本，则返回的是多个样本被预处理之后的结果list。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-b70jh26IrJS",
        "outputId": "fc238b7f-a423-4fdf-f896-df540af1bec3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': [[101, 7327, 19164, 2446, 2655, 2000, 17757, 2329, 12559, 1012, 102], [101, 2848, 13934, 102], [101, 9371, 2727, 1011, 5511, 1011, 2570, 102], [101, 1996, 2647, 3222, 2056, 2006, 9432, 2009, 18335, 2007, 2446, 6040, 2000, 10390, 2000, 18454, 2078, 2329, 12559, 2127, 6529, 5646, 3251, 5506, 11190, 4295, 2064, 2022, 11860, 2000, 8351, 1012, 102], [101, 2762, 1005, 1055, 4387, 2000, 1996, 2647, 2586, 1005, 1055, 15651, 2837, 14121, 1062, 9328, 5804, 2056, 2006, 9317, 10390, 2323, 4965, 8351, 4168, 4017, 2013, 3032, 2060, 2084, 3725, 2127, 1996, 4045, 6040, 2001, 24509, 1012, 102]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], 'labels': [[-100, 3, 0, 7, 0, 0, 0, 7, 0, 0, -100], [-100, 1, 2, -100], [-100, 5, 0, 0, 0, 0, 0, -100], [-100, 0, 3, 4, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -100], [-100, 5, 0, 0, 0, 0, 0, 3, 4, 0, 0, 0, 0, 1, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, -100]]}"
            ]
          },
          "execution_count": 23,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenize_and_align_labels(datasets['train'][:5])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zS-6iXTkIrJT"
      },
      "source": [
        "接下来对数据集datasets里面的所有样本进行预处理，处理的方式是使用map函数，将预处理函数prepare_train_features应用到（map)所有样本上。\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DDtsaJeVIrJT"
      },
      "outputs": [],
      "source": [
        "tokenized_datasets = datasets.map(tokenize_and_align_labels, batched=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "voWiw8C7IrJV"
      },
      "source": [
        "更好的是，返回的结果会自动被缓存，避免下次处理的时候重新计算（但是也要注意，如果输入有改动，可能会被缓存影响！）。datasets库函数会对输入的参数进行检测，判断是否有变化，如果没有变化就使用缓存数据，如果有变化就重新处理。但如果输入参数不变，想改变输入的时候，最好清理调这个缓存。清理的方式是使用`load_from_cache_file=False`参数。另外，上面使用到的`batched=True`这个参数是tokenizer的特点，以为这会使用多线程同时并行对输入进行处理。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "545PP3o8IrJV"
      },
      "source": [
        "## 微调预训练模型"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FBiW8UpKIrJW"
      },
      "source": [
        "\n",
        "既然数据已经准备好了，现在我们需要下载并加载我们的预训练模型，然后微调预训练模型。既然我们是做seq2seq任务，那么我们需要一个能解决这个任务的模型类。我们使用`AutoModelForTokenClassification` 这个类。和tokenizer相似，`from_pretrained`方法同样可以帮助我们下载并加载模型，同时也会对模型进行缓存，就不会重复下载模型啦。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 154,
          "referenced_widgets": [
            "d8e51d8a73cd43e7a2a8df46ec8235fd",
            "9cd2aab849df4c67b9808376ce72a4d6",
            "a4fea0605a4545ba9f8d91de8c220d5b",
            "f07db88e03144d09bff802d1cc8b49eb",
            "4018693393dc432c89c22af62db4adc8",
            "065dc405d3ea4a54884042a7e840cd2d",
            "457aae4f3187403a95250a20ca64efda",
            "987a3d54e903459c8243fdf623897a8e",
            "c4f56d53da3d4297a9cfc6001a3364d8",
            "41a0b972491d464a8477fb1af243a652",
            "83294cf46f544a27a98ea79fd0bc3834"
          ]
        },
        "id": "TlqNaB8jIrJW",
        "outputId": "552d8540-8a2a-4def-9563-b4942e9bd0ce"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d8e51d8a73cd43e7a2a8df46ec8235fd",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForTokenClassification: ['vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_projector.bias', 'vocab_projector.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight']\n",
            "- This IS expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing DistilBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of DistilBertForTokenClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        }
      ],
      "source": [
        "from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer\n",
        "\n",
        "model = AutoModelForTokenClassification.from_pretrained(model_checkpoint, num_labels=len(label_list))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CczA5lJlIrJX"
      },
      "source": [
        "由于我们微调的任务是token分类任务，而我们加载的是预训练的语言模型，所以会提示我们加载模型的时候扔掉了一些不匹配的神经网络参数（比如：预训练语言模型的神经网络head被扔掉了，同时随机初始化了token分类的神经网络head）。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_N8urzhyIrJY"
      },
      "source": [
        "\n",
        "\n",
        "为了能够得到一个`Trainer`训练工具，我们还需要3个要素，其中最重要的是训练的设定/参数 [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments)。这个训练设定包含了能够定义训练过程的所有属性。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Bliy8zgjIrJY"
      },
      "outputs": [],
      "source": [
        "args = TrainingArguments(\n",
        "    f\"test-{task}\",\n",
        "    evaluation_strategy = \"epoch\",\n",
        "    learning_rate=2e-5,\n",
        "    per_device_train_batch_size=batch_size,\n",
        "    per_device_eval_batch_size=batch_size,\n",
        "    num_train_epochs=3,\n",
        "    weight_decay=0.01,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "km3pGVdTIrJc"
      },
      "source": [
        "上面evaluation_strategy = \"epoch\"参数告诉训练代码：我们每个epcoh会做一次验证评估。\n",
        "\n",
        "上面batch_size在这个notebook之前定义好了。\n",
        "\n",
        "最后我们需要一个数据收集器data collator，将我们处理好的输入喂给模型。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dOEjMjQiIBu_"
      },
      "outputs": [],
      "source": [
        "from transformers import DataCollatorForTokenClassification\n",
        "\n",
        "data_collator = DataCollatorForTokenClassification(tokenizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kj5MUkECIBu_"
      },
      "source": [
        "\n",
        "设置好`Trainer`还剩最后一件事情，那就是我们需要定义好评估方法。我们使用[`seqeval`](https://github.com/chakki-works/seqeval) metric来完成评估。将模型预测送入评估之前，我们也会做一些数据后处理：\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "abNA0xxBIBu_"
      },
      "outputs": [],
      "source": [
        "metric = load_metric(\"seqeval\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6IGgvU5QIBu_"
      },
      "source": [
        "评估的输入是预测和label的list"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RXh2lLfzIBu_",
        "outputId": "660c2c05-fd3b-4ee4-c394-f5b20b3626c3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'LOC': {'f1': 1.0, 'number': 2, 'precision': 1.0, 'recall': 1.0},\n",
              " 'ORG': {'f1': 1.0, 'number': 1, 'precision': 1.0, 'recall': 1.0},\n",
              " 'PER': {'f1': 1.0, 'number': 1, 'precision': 1.0, 'recall': 1.0},\n",
              " 'overall_accuracy': 1.0,\n",
              " 'overall_f1': 1.0,\n",
              " 'overall_precision': 1.0,\n",
              " 'overall_recall': 1.0}"
            ]
          },
          "execution_count": 30,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "labels = [label_list[i] for i in example[f\"{task}_tags\"]]\n",
        "metric.compute(predictions=[labels], references=[labels])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7sZOdRlRIrJd"
      },
      "source": [
        "对模型预测结果做一些后处理：\n",
        "- 选择预测分类最大概率的下标\n",
        "- 将下标转化为label\n",
        "- 忽略-100所在地方\n",
        "\n",
        "下面的函数将上面的步骤合并了起来。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UmvbnJ9JIrJd"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "def compute_metrics(p):\n",
        "    predictions, labels = p\n",
        "    predictions = np.argmax(predictions, axis=2)\n",
        "\n",
        "    # Remove ignored index (special tokens)\n",
        "    true_predictions = [\n",
        "        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
        "        for prediction, label in zip(predictions, labels)\n",
        "    ]\n",
        "    true_labels = [\n",
        "        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
        "        for prediction, label in zip(predictions, labels)\n",
        "    ]\n",
        "\n",
        "    results = metric.compute(predictions=true_predictions, references=true_labels)\n",
        "    return {\n",
        "        \"precision\": results[\"overall_precision\"],\n",
        "        \"recall\": results[\"overall_recall\"],\n",
        "        \"f1\": results[\"overall_f1\"],\n",
        "        \"accuracy\": results[\"overall_accuracy\"],\n",
        "    }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rXuFTAzDIrJe"
      },
      "source": [
        "我们计算所有类别总的precision/recall/f1，所以会扔掉单个类别的precision/recall/f1 \n",
        "\n",
        "将数据/模型/参数传入`Trainer`即可\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "imY1oC3SIrJf"
      },
      "outputs": [],
      "source": [
        "trainer = Trainer(\n",
        "    model,\n",
        "    args,\n",
        "    train_dataset=tokenized_datasets[\"train\"],\n",
        "    eval_dataset=tokenized_datasets[\"validation\"],\n",
        "    data_collator=data_collator,\n",
        "    tokenizer=tokenizer,\n",
        "    compute_metrics=compute_metrics\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CdzABDVcIrJg"
      },
      "source": [
        "调用`train`方法开始训练"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3VFegC_2IBvA",
        "outputId": "04a00784-c591-4c36-b923-b6c631d82e97"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "        </style>\n",
              "      \n",
              "      <progress value='2634' max='2634' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [2634/2634 01:45, Epoch 3/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "      <th>Precision</th>\n",
              "      <th>Recall</th>\n",
              "      <th>F1</th>\n",
              "      <th>Accuracy</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.237721</td>\n",
              "      <td>0.068198</td>\n",
              "      <td>0.903148</td>\n",
              "      <td>0.921132</td>\n",
              "      <td>0.912051</td>\n",
              "      <td>0.979713</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>0.053160</td>\n",
              "      <td>0.059337</td>\n",
              "      <td>0.927697</td>\n",
              "      <td>0.932990</td>\n",
              "      <td>0.930336</td>\n",
              "      <td>0.983113</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.029850</td>\n",
              "      <td>0.059346</td>\n",
              "      <td>0.929267</td>\n",
              "      <td>0.939143</td>\n",
              "      <td>0.934179</td>\n",
              "      <td>0.984257</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=2634, training_loss=0.08569671253227518)"
            ]
          },
          "execution_count": 33,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CKASz-2vIrJi"
      },
      "source": [
        "我们可以再次使用`evaluate`方法评估，可以评估其他数据集。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UOUcBkX8IrJi",
        "outputId": "de5b9dd6-9dc0-4702-cb43-55e9829fde25"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "        <style>\n",
              "            /* Turns off some styling */\n",
              "            progress {\n",
              "                /* gets rid of default border in Firefox and Opera. */\n",
              "                border: none;\n",
              "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "                background-size: auto;\n",
              "            }\n",
              "        </style>\n",
              "      \n",
              "      <progress value='408' max='204' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [204/204 00:05]\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "{'eval_loss': 0.05934586375951767,\n",
              " 'eval_precision': 0.9292672127518264,\n",
              " 'eval_recall': 0.9391430808815304,\n",
              " 'eval_f1': 0.9341790463472988,\n",
              " 'eval_accuracy': 0.9842565968195466,\n",
              " 'epoch': 3.0}"
            ]
          },
          "execution_count": 34,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.evaluate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lpG43ghrIBvB"
      },
      "source": [
        "如果想要得到单个类别的precision/recall/f1，我们直接将结果输入相同的评估函数即可："
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ozldLGLVIBvB",
        "outputId": "6f4a8b25-16a1-4098-b455-7fac3d2440df"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'LOC': {'precision': 0.949718574108818,\n",
              "  'recall': 0.966768525592055,\n",
              "  'f1': 0.9581677077418134,\n",
              "  'number': 2618},\n",
              " 'MISC': {'precision': 0.8132387706855791,\n",
              "  'recall': 0.8383428107229894,\n",
              "  'f1': 0.8255999999999999,\n",
              "  'number': 1231},\n",
              " 'ORG': {'precision': 0.9055232558139535,\n",
              "  'recall': 0.9090466926070039,\n",
              "  'f1': 0.9072815533980583,\n",
              "  'number': 2056},\n",
              " 'PER': {'precision': 0.9759552042160737,\n",
              "  'recall': 0.9765985497692815,\n",
              "  'f1': 0.9762767710049424,\n",
              "  'number': 3034},\n",
              " 'overall_precision': 0.9292672127518264,\n",
              " 'overall_recall': 0.9391430808815304,\n",
              " 'overall_f1': 0.9341790463472988,\n",
              " 'overall_accuracy': 0.9842565968195466}"
            ]
          },
          "execution_count": 35,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "predictions, labels, _ = trainer.predict(tokenized_datasets[\"validation\"])\n",
        "predictions = np.argmax(predictions, axis=2)\n",
        "\n",
        "# Remove ignored index (special tokens)\n",
        "true_predictions = [\n",
        "    [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
        "    for prediction, label in zip(predictions, labels)\n",
        "]\n",
        "true_labels = [\n",
        "    [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
        "    for prediction, label in zip(predictions, labels)\n",
        "]\n",
        "\n",
        "results = metric.compute(predictions=true_predictions, references=true_labels)\n",
        "results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vuvfNwpIBvB"
      },
      "source": [
        "最后别忘了，查看如何上传模型 ，上传模型到](https://huggingface.co/transformers/model_sharing.html) 到[🤗 Model Hub](https://huggingface.co/models)。随后您就可以像这个notebook一开始一样，直接用模型名字就能使用您自己上传的模型啦。\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7w_OhlOpIBvB"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "n9qywopnIrJH",
        "545PP3o8IrJV",
        "7k8ge1L1IrJk"
      ],
      "name": "4.2-序列标注",
      "provenance": []
    },
    "interpreter": {
      "hash": "3bfce0b4c492a35815b5705a19fe374a7eea0baaa08b34d90450caf1fe9ce20b"
    },
    "kernelspec": {
      "display_name": "Python 3.8.10 64-bit ('venv': virtualenv)",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": ""
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "065dc405d3ea4a54884042a7e840cd2d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4018693393dc432c89c22af62db4adc8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_83294cf46f544a27a98ea79fd0bc3834",
            "placeholder": "​",
            "style": "IPY_MODEL_41a0b972491d464a8477fb1af243a652",
            "value": " 268M/268M [00:05&lt;00:00, 54.1MB/s]"
          }
        },
        "41a0b972491d464a8477fb1af243a652": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "457aae4f3187403a95250a20ca64efda": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "83294cf46f544a27a98ea79fd0bc3834": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "987a3d54e903459c8243fdf623897a8e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "9cd2aab849df4c67b9808376ce72a4d6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a4fea0605a4545ba9f8d91de8c220d5b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_457aae4f3187403a95250a20ca64efda",
            "placeholder": "​",
            "style": "IPY_MODEL_065dc405d3ea4a54884042a7e840cd2d",
            "value": "Downloading: 100%"
          }
        },
        "c4f56d53da3d4297a9cfc6001a3364d8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d8e51d8a73cd43e7a2a8df46ec8235fd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a4fea0605a4545ba9f8d91de8c220d5b",
              "IPY_MODEL_f07db88e03144d09bff802d1cc8b49eb",
              "IPY_MODEL_4018693393dc432c89c22af62db4adc8"
            ],
            "layout": "IPY_MODEL_9cd2aab849df4c67b9808376ce72a4d6"
          }
        },
        "f07db88e03144d09bff802d1cc8b49eb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c4f56d53da3d4297a9cfc6001a3364d8",
            "max": 267967963,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_987a3d54e903459c8243fdf623897a8e",
            "value": 267967963
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}